SCHOOL-TO-WORK 
INTERMEDIARY PROJECT 



Data AND Evidence Gathering: 
Strategies and Challenges 

By Hilary Kopp 

During public debates over the efficacy of school-to-career, issues of evidence and data took 
center stage. Did school-to-career make a difference for young people? For which young people? 
How should progress in building school-to-career systems and programs be measured? Frustra- 
tion with the difficulty of demonstrating gains in academic achievement became a frequent topic 
of discussion by practitioners and policymakers alike, even as proponents argued that other 
measures, such as employer evaluations or rates for college-going and attendance, could provide 
important evidence of student and system progress. 

For local intermediary organizations that perform key connecting and brokering functions 
linking schools, employers, and young people, data and evidence debates are even more compli- 
cated. How does an organization that convenes key stakeholders measure its effectiveness? If 
an intermediary helps set up internships, should its usefulness be measured by student achieve- 
ment, by employer and school satisfaction, or by other measures of value-added? On what evi- 
dence should an intermediary judge its own work— and why? 



When the School-to-Work Intermediary 
Project began in October 1998 most members 
of the emerging Intermediary Network were 
engaged only marginally in thinking about the 
best ways to measure their own effectiveness.^ 
They were working to get school-to-career pro- 
grams and activities up and running. If they 
focused on data and evidence, it was typically in 
service of supporting the community's or 
region’s school-to-career system, not of improv- 
ing their own effectiveness or demonstrating 
their value to outside observers, such as fun- 
ders, political leaders, or critics. 

Over the years. Network members' 
views on data and evidence have changed 
markedly. Many began to see credible evi- 
dence as a critical element of both their 
approaches to program improvement and 
their strategies for winning and sustaining sup- 
port and resources. Network members have 
taken a hard look at how best to use data and 



evidence in support of intermediary activities 
and initiatives. 

This issue brief: 

• Describes the activities Network members 
have pursued together during the past three 
years that addressed data and evidence con- 
cerns; and 

• Draws lessons from the efforts of Network 
members, as a group and individually, as they 
have sought to design and implement data 
and evidence-gathering systems that could 
help them make their case to others and 
improve their performance over time. 

Collective Efforts to Address 
Evidence of Intermediary Value 

In its second year, the Intermediary Network 
created a Data Work Group, a volunteer sub- 
set of Network members who were interested 
in addressing data and evidence issues facing 
their organizations. This group shared data col- 
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lection tools and strategies and discussed 
design and Implementation challenges. 

The Data Work Group identified several 
promising strategies In use or in development 
by organizations in and outside the Network. 
For example, BaySCAN In California had 
asked Its Bay Area local partnerships to fill out 
a common data form Identifying the numbers 
of schools, students, teachers, and workforce 
partners Involved In school-to-career activi- 
ties. BaySCAN made each of these data 
sheets available on Its Web site, www. 
bayscan.org. Workforce Silicon Valley, a part- 
ner of BaySCAN and also of the recently 
formed statewide group CalSCAN, devel- 
oped a template for assigning dollar equiva- 
lents to the In-kind contributions of business 
partners, yielding estimates of the cost to 
employers of activities ranging from job shad- 
ows to teacher training, board membership, 
and student internships. This template en- 
abled organizations to generate rough "order- 
of-magnitude" calculations of the business 
investment in school-to-career efforts. To 
access the template, go to: www.CalSCAN.org. 

At the same time, the Data Working 
Group began to address practical ways for 
intermediary organizations to assess their 
own activities and present evidence of their 
effectiveness. New Ways to Work, on behalf 
of the Intermediary Project, created self- 



Proposed Measures for Demonstrating Intermediary Value 



Level of Activity (total numbers and 
change over time) 

•Youth served 

• Placements In work-based 
learning 

• Employers Involved 

• Schools participating 

Customer Satisfaction 

•Surveys of students, teachers, and 
employers 

Leveraging of Resources 

•Wages paid or cash value of 
Imputed wages for students in 
work-based learning 



Educational Success 
•College-going rate 

• Graduation/Drop-out rate 

• Pass rate for state assessment test 
(initial rate and rate after retest) 

Employment Success 

■ Quality of employment after 
graduation (e.g., wages, benefits, 
training opportunities) 



assessment tools and encouraged organiza- 
tions to use these to define goals and progress 
toward goals related to the four functions of 
local intermediaries: convening partners, pro- 
viding or brokering services, measuring 
progress and ensuring quality, and advocating 
for favorable policies.^ 

With support from Jobs for the Future, 
the Data Work Group began to define what 
it thought would be useful measures of Inter- 
mediary effectiveness. A complex matrix of 
Indicators was developed to capture the rich- 
ness and value of effective Intermediaries 
across the four Intermediary functions. How- 
ever, the complexity of initial drafts of such a 
matrix made it clear that ambition out- 
stripped the ability of any intermediary in the 
Network to collect all the desired data. 

In response to this dilemma. Network 
members agreed upon a more modest list of 
useful data elements at ajune 2001 Network 
meeting. The list (see box) was a composite of 
evidence that has been collected by various 
Network members, including the Boston Pri- 
vate Industry Council. It Identified five broad 
categories where Network members felt evi- 
dence could bolster the case for Intermediary 
support: 

• The level of activity, defined as numbers of stu- 
dents, schools, and workplaces participating; 

• The level of satisfaction of key customers. 
Including employers, schools, and students; 

• The dollar value of wages of students partici- 
pating In work-based learning, either actual 
or imputed; 

• Educational outcomes, with an eye toward 
measures beyond scores on standardized 
tests; and 

• Employment success, particularly for those 
seeking full- or part-time employment after 
graduation. 

Network members recognized that only 
some ofthe measures proposed for collection 
directly captured the effect of an Intermedi- 
ary. Customer service surveys could be field- 
ed to stakeholders convened or served by the 
Intermediary. The number of placements and 
other indications of level of activity might be 




attributable to the intermediaries' work, but 
such measures could also capture the efforts 
of schools or other local players. The same 
problem reappears when considering student 
outcome data. What role did the intermedi- 
ary play in affecting student academic 
achievement, attendance, or college-going? 

In fall 2001, Jobs for the Future surveyed 
the 50 Intermediary Network members to 
ascertain the kinds of data and evidence each 
organization was collecting in some form or 
another. Over 20 sites responded to the sur- 
vey. Many reported collecting a good deal of 
input data (i.e., the level of activity and 
involvement of different partners). Fewer col- 
lected meaningful data on student outcomes, 
particularly on post-graduation educational or 
employment outcomes. 

Later in the fall. Network members dis- 
cussed the potential for a collective effort that 
would result in evidence collated across multiple 
sites, giving a picture of the reach and scope of 
Network activities nationwide. At that meeting, 
the methodological challenges (e.g., defining 
what was being measured in ways that did not 
compare apples to oranges) appeared too 
daunting to many Network members, particu- 
larly those with modest resources for evidence 
gathering and analysis. 

In the end, participants in the meeting 
agreed to collect three simple kinds of data: 
the number of students involved; the number 
of schools involved; and the number of 
employers involved. Thirteen intermediaries 
supplied this information (see Appendix). The 
data shows the range of communities in which 
the intermediaries are working (from single, 
large districts to many school districts within 
a region) and the variety of school-to-career 
activities they support. 

Lessons on the Challenges 
AND Strategies for Gathering 
Evidence: Experiences of a 
Sample of Intermediaries 

In spring 2002, staff from 12 intermediary 
organizations active in the Data Working 
Group were interviewed at length concerning 



their data collection and evaluation practices. 
These organizations have; 

• Made varying degrees of progress in imple- 
menting data and evidence-gathering sys- 
tems; 

• Developed several successful strategies for 
identifying relevant evidence and collecting 
useful data; and 

• Encountered a number of significant chal- 
lenges. 

Data and evidence collection 
is a growing priority. 

In general, data collection and the need for evi- 
dence have taken on a higher priority for 
school-to-career intermediaries over the 
years. Yet identifying appropriate outcomes to 
measure is not straightforward when so much 
of an intermediary's work focuses on develop- 
ing new partnerships and contributing to 
changes in how schools educate children. 
Thinking through the logic of an intermediary's 
value-added and how those results can be 
measured are challenging evaluation questions. 

Over the past five or more years, many 
Network members have evolved from count- 
ing the number of participants (e.g., students, 
teachers, employers) in different activities to 
assessing, through systematic surveys and 
other feedback mechanisms, the quality of the 
intermediaries' direct interventions and sup- 
port for partners. Some organizations have 
started to look at the impact of their activities 
on young people's school performance, 
despite the difficulties of measuring broader 
community impacts. And as federal School-to- 
Work funding ends, securing evidence on the 
larger community impact has proven increas- 
ingly important to securing sustainable funding. 

Staff of the intermediary organizations 
interviewed indicated that limited resources 
for evaluation and lack of agreement on appro- 
priate measures have made it difficult to clear- 
ly define and gather evidence. Most intermedi- 
aries have multiple funders who want data and 
evidence on different things. Some intermedi- 
aries work with multiple local partnerships 
that are interested in different outcomes. Sev- 
eral intermediary staff indicated that they felt 



there was little guidance from their state 
and/or federal funders concerning what indi- 
cators they should be collecting data on. Some 
states have been inconsistent about the data 
they require or unable to share data with local 
or regional partnerships in a usable form. 

This lack of consensus is not surprising. 
Local partnerships have had significant auton- 
omy, and each intermediary plays a somewhat 
unique role in its community. Each intermedi- 
ary has had to figure out data and evidence 
collection largely on its own. Given chal- 
lenges— of cost, complexity, and, frequently, 
insufficient capacity to design and implement 
efficient evidence-gathering systems— data col- 
lection does not always get to the top of the 
“to-do" list until a funder demands more evi- 
dence. Unfortunately, retroactive data collec- 
tion and analysis are even more difficult than 
building in data-gathering procedures from 
the outset. 

Some intermediaries have successfully 
built in data collection and feedback mecha- 
nisms when creating new programs and initia- 
tives. Barbara Duffy, executive director of MY 
TURN, Inc., in Brockton, Massachusetts, 
believes that intermediaries need to get cre- 
ative about how to collect information and 
feedback on what they are doing as soon as 
possible: "If you wait five years to do any evalu- 
ation, you won’t be around by the end of the 
five years." MY TURN program staff are 
responsible for tracking student participation 
and outcomes and for designing questionnaires 
to get feedback from participants on their 
experience with the program and what could 
be changed. MY TURN has also tapped into 
local college and university resources to get 
low-cost evaluation work done by third parties. 

Richard Tulikangas of Linking Learning to 
Life in Burlington, Vermont, believes that 
"what gets measured, gets done." He has made 
an effort to articulate what the organization 
should be measuring on an ongoing basis to 
assure that the intermediary contributes to 
these outcomes. Linking Learning to Life holds 
quarterly "action research days” to enable staff 
to step back from their day-to-day work to 



look at data, discuss progress in the work, look 
at progress against their strategic plan, and 
determine what additional information or data 
they might need to collect. 

Both of these intermediaries have creat- 
ed a culture of reflection that supports ongo- 
ing evaluation. Evaluations can make people 
nervous, and negative feedback is not always 
easy to handle. Yet when the director consis- 
tently models openness to feedback and a 
desire to know how the organization is doing, 
staff members seem to follow this lead. It 
helps if staff responsibilities include some data 
collection: this makes it clear that getting 
information on how the work is going is part 
of the job. 

Two common approaches are tracking 
participation levels and getting 
“customer” feedback through surveys 
and focus groups. 

Intermediary Network members often under- 
take and support a mix of activities and initia- 
tives. The ease with which an intermediary 
can collect data and evidence on levels of par- 
ticipation, “customer satisfaction," and out- 
comes relates directly to the types of activities 
the organization undertakes. Intermediaries 
that do not have youth as direct “clients” 
depend heavily on their education and busi- 
ness partners to collect any type of data. 

Data collection is a simpler task for pro- 
grams directly operated by the intermediary 
because intermediary staff interact directly 
with students, teachers, or employers. Inter- 
mediaries that support school-to-career pro- 
gramming in schools (e.g., career academies, 
pathways) depend upon their school partners 
to provide data on the numbers of people 
involved— as well as the kind and intensity of 
involvement. The number of students benefit- 
ing from less structured, intermediary-sup- 
ported activities (e.g., career centers and 
career fairs, business people coming into the 
classroom supporting the work of school- 
based school-to-career coordinators) is harder 
to capture and quantify. Students may partici- 
pate in each of these efforts more than once. 
Moreover, matching activities to important 



outcomes for students is no straightforward 
manner. 

All of the intermediary organizations 
interviewed collect information on the num- 
bers of different types of people or partners 
involved in their work. This type of data is 
important for reporting to funders as well as 
for providing a sense of the overall level of 
school-to-career programming put in place by 
the intermediary and its partners. Such data, 
especially if tracked over time, also gives an 
indication of the extent to which school-to- 
career interventions are growing within a 
community and “touching” more students, 
employers, schools, or teachers. 

Many intermediaries use surveys and 
focus groups to gather feedback from their 
partners and “customers" on the school-to- 
career activities. “Customer satisfaction" sur- 
veys are a relatively quick and inexpensive 
way to get information on how such activities 
are being received and to get specific infor- 
mation on elements that need fine-tuning. Sur- 
veys can also provide data to report to fun- 
ders: responses can be aggregated for findings 
such as "90 percent of the students who par- 
ticipated in a work internship felt the experi- 
ence was worthwhile.” Many intermediary 
staff reported that feedback on the quality of 
their services and the support they offer to 
their partners is the most useful data to them 
in terms of improvingthe work they do. 

Most commonly, intermediaries use short 
questionnaires to get feedback from students, 
employers, and teachers involved in work- 
place-based experiences, such as job shadows 
and internships. For example, the Pittsburgh 
Technology Council has developed an employ- 
er survey that asks for very specific feedback 
from employers participating in its Manufac- 
turing Pathway Initiative to inform program 
adjustments. Intermediaries who do profes- 
sional development for teachers or provide 
teaching resources also ask for feedback on 
their provision of these services and 
resources. To access Pittsburgh Technology Coun- 
cil's survey, go to: www.intermediarynetwork.org. 

A smaller number of intermediaries 



reported using surveys and focus groups to 
get partner feedback on the general work and 
focus of the intermediary organization itself 
Such feedback can be useful for assessing how 
well the organization is connecting and sup- 
porting its partners, as well as for eliciting 
feedback on whether to continue current 
activities and what new ideas to pursue. 
While intermediaries typically meet regularly 
with their partners, questions on how the 
partnership is doing can be more honestly 
answered when a neutral outsider facilitates a 
focus group or conducts a series of interviews. 
Intermediaries that have conducted surveys 
and focus groups report that the feedback is 
very useful for strategic planning. ("See box, 
"Getting Feedback on How Well the Partnership 
is Working. ") 

Strategies to consider when developing 
"customer satisfaction” surveys; 

Surveys or questionnaires can also offer 
an opportunity for intermediaries to gather 
information that goes beyond feedback on a 
specific experience or activity. For example, 
questionnaires can include background ques- 



Getting Feedback on How Well the Partnership is Working 



The Workplace Learning Connection in Cedar Rapids, Iowa, hired a 
retired school-to-career coordinator to interview education partners. Using 
a structured questionnaire, the coordinator asked the partners about what 
activities they thought the Workplace Learning Connection should continue 
to do, stop doing, or start doing. Their level of participation in school-to- 
career activities (and how they rated them) was also documented. A few 
business partners were tapped to do a similar series of interviews with other 
business partners. To access the questionnaires, go to: 
www.intermediarynetwork.org. 



MYTURN, Inc., in Brockton, Massachusetts, periodically holds breakfast 
focus groups with different groups of its partners. An outside facilitator runs 
these meetings. Partners are asked about their perceptions of MYTURN, 
what they liked and disliked about recent MYTURN activities, what the 
partners were interested in seeing MY TURN accomplish, and how MY 
TURN could better support and engage partners. 



To help guide the focus of its future work, the Tulare County, California, 
School-to-Career Office is surveying its partners on the activities they 
think are most important to future school-to-career or business-education 
efforts. The questionnaire offers a range of work-based, school-based, and 
connecting activities to consider (and encourages partners to suggest new 
ones). To access the questionnaires, go to: www.intermediarynetwork.org. 





tions on participants so intermediaries can 
better understand and describe who they 
serve. Questions can ask about how partici- 
pants found out about the program and why 
they decided to participate. Such information 
can help inform marketing and outreach 
efforts. 

Surveys can also ask participants how 
they think they will use the knowledge or 
experience they gained in the future. For 
example, Linking Learning to Life’s post- 
internship survey for teachers asks about 
ways the experience might be translated to 
activities during the academic year and probes 
the likelihood that the teacher will use any of 
these methods or activities. The Workplace 
Learning Connection asks students who have 
participated in an internship whether they will 



Using Survey and Other Data for Needs Assessment Purposes 



Pittsburgh Technology Council: As a first step in creating a professional 
development effort on information technology literacy, the Pittsburgh Tech- 
nology Council surveyed teachers about what they were doing with informa- 
tion technology in their classes and about their comfort level with the IT 
skills standards identified by employers as important for all workers. The 
council used this assessment to determine what information technology- 
related course each teacher should take at the University of Pittsburgh, as 
well as the focus of additional workshops. A post-assessment survey is 
planned to see if teachers have changed the extent to which they incorpo- 
rate IT into the classroom. 



In an effort to document what schools were doing and how that compared 
to what employers need, the council arranged for large-scale testing of a ran- 
dom sample of seniors in the nine-county region where it targets its 
programs. Students took the WorkKeys test developed by ACT to assess 
foundational skill areas using criteria identified by employers (e.g., reading 
for information, applied math, graphic interpretation). When the results 
were compared to a national profile of what people need for sustainable 
employment, 40 percent of the students were found to have limited skills to 
the point that they would have little job mobility. 

Tulare County Workforce Coalition: Randy Wallace from Tulare County 
believes that “data helps to generate conversations." He is using focus groups 
and surveys with students to find out what makes a difference in young peo- 
ple’s learning. This effort will enable the Tulare County Workforce Coalition 
to compare responses and experiences of students in the career pathways 
and academies they support to those of students who are not in special 
career-related programs. The intermediary is also looking beyond this com- 
parison to see what kinds of opportunities and experiences help to motivate 
and direct young people. This information will inform both the intermedi- 
ary’s and the school districts' programming. 



consider pursuing a career in the same field, 
whether they have listed the experience on 
scholarship or job applications (and if they 
think the internship experience has helped 
them achieve any of these goals), and if they 
would recommend the internship experience 
to a friend. To access the survey instruments, go 
to: www.intermediarynetwork.org. 

Surveys can also assess needs or test out 
how people will respond to a new program or 
intervention. However, few of the intermedi- 
aries interviewed use surveys for needs 
assessment. Data collection on the front-end 
can help shape programming and profession- 
al development as well as help to make a case 
for a particular intervention. (See box, “Using 
Survey and Other Data for Needs Assessment 
Purposes.”) 

When developing customer-satisfaction 
questionnaires, intermediaries may want to 
phrase some questions in a close-ended for- 
mat, with answer options that the respondent 
can check off. This makes it easier for the 
respondent to complete the questionnaire 
(which can lead to higher response rates) and 
for intermediary staff to tabulate and report 
results. When surveys include only open- 
ended questions, it can be difficult to aggre- 
gate the results for reporting findings and for 
assessing what participants have to say about 
the program. 

Some Efforts Being Made to 
Gather Evidence on the Impact 
OF School-to-Career on 
Students and Community 

As discussed above, the pressure and desire 
are increasing to gather more evidence on 
whether school-to-career activities and inter- 
ventions are making a difference, and what 
impact or contribution an intermediary makes 
in a community. Some Intermediary Network 
members have brought in outside evaluators 
to help them tackle these complex questions. 
This section highlights the experiences of four 
intermediaries in gathering and analyzing evi- 
dence on the impact of school-to-career and 
their activities. 




Tulare County Workforce Coalition, 
Visalia, California: The Tulare County 
Workforce Coalition has worked in partner- 
ship with an outside evaluator and several 
school districts to evaluate well-defined 
school-to-career initiatives. These evaluation 
efforts strive to find evidence that school-to- 
career is resulting in positive effects on stu- 
dents and to gain insight on what makes a dif- 
ference in young people’s learning. The 
evaluation work utilizes school records data 
on attendance, GPA, test scores, dropout 
rates, and transition to further education or 
careers after graduating from high school. 
Outcomes for students in more intensive 
school-to-career activities (such as Career 
Academies) are compared to those of the 
general school population, and changes in out- 
come indicators overtime are examined for 
the whole school population. In addition, eval- 
uators field a survey to a broad sample of stu- 
dents that includes both Academy and non- 
Academy participants, and responses for the 
two groups are compared. Focus groups are 
also held with a mix of students. 

A recent analysis of the impact of school- 
to-career activity in Golden West High School 
found an increase in positive outcomes for 
Career Academy and Pathway participants 
relative to the general school population.^ 
Pathway and Academy completers experi- 
enced a 6.6 percent increase in GPA com- 
pared to overall student GPAs during the 
same period. Gompared to the general school 
population. Health Career Academy com- 
pleters were 47 percent more likely to have 
interned or worked in a field related to their 
area of interest and 30 percent more likely to 
plan to pursue a college degree (based on self- 
reported data). Several positive trends in the 
overall school population’s test scores, reten- 
tion rate, and community college transition 
rate were also found. 

Philadelphia Youth Network, Philadel- 
phia, Pennsylvania: The Philadelphia Youth 
Network started as an in-district school-to- 
career/education reform office. Until it 
became independent of the school district, it 



was well-positioned to access school records 
data to assess the impact of school-to-career 
activities on participating students and schools 
at large. Over the years, the Philadelphia 
Youth Network has done a number of evalua- 
tions directly and brought in the assistance of 
outside evaluators. It found that academic per- 
formance, test scores, and graduation rates 
improved across the district as school-to- 
career and other education reform efforts 
were more fully implemented (although 
results could not be solely attributed to 
school-to-career interventions). A study by an 
outside evaluator examined school perform- 
ance outcomes in four high schools and found 
some positive changes associated with school- 
to-career participation. 

Structured work-based learning experi- 
ences have been a major component of 
school-to-career activities in Philadelphia. Sev- 
eral studies have been conducted on the 
impact of work-based learning and all found 
positive effects.^ One study by an outside 
evaluator compared participants in work- 
based learning to non-participants who were 
also deemed “work-ready” at four high 
schools. The analysis found that participation 
in work-based learning significantly improved 
GPAs and attendance. (The analysis con- 
trolled for the influence of prior grades, atten- 
dance, year in school, and school attended.) 
Longer participation in work-based learning 
had more significant, positive effects than 
shorter participation. 

The school district also analyzed school 
records data for work-based learning partici- 
pants compared to similar data for their non- 
participating peers citywide. This analysis 
showed higher attendance and graduation 
rates, lower dropout rates, and better grades 
for participants in work-based learning. These 
positive results held true when the scope of 
analysis was narrowed to only those students 
in small learning communities who had similar 
grades and attendance at the time they 
entered a small learning community. (Anoth- 
er study, examining the impact of work-based 
learning participation, was conducted via a 



telephone survey of recent alumni and is 
described below.) 

Business/Education Expectations, Kansas 
City, Missouri: The BE^: School-to-Career 
Partnership is a bi-state organization managed 
by the Learning Exchange. Early In 1999, the 
partnership decided to undertake an exten- 
sive, 30-month evaluation effort to examine 
the support for school-to-career in the 
Greater Kansas City metropolitan area, estab- 
lish the organization’s credibility as a driving 
force in building school-to-career, and attain 
evidence for Its sustainability. The evaluation 
sought to address two major questions: What 
is the worth of school-to-career as a school 
reform intervention? And what is the worth 
or usefulness of the BE^: School-to-Career 
Partnership as a mechanism in supporting the 
development and growth of school-to-career? 

The evaluation essentially Involved four 
separate studies designed to address different 
pieces of these questions. Surveys, interviews, 
focus groups, and an analysis of school 
records data (for high school students only) 
were used to look at how schools were imple- 
menting school-to-career activities and the 
impact of school-to-career on young people. 
Telephone interviews with employers, par- 
ents, and educators were used to capture the 
perceptions of stakeholders concerning 
school-to-career generally. Interviews with 
partnership staff, board members, and func- 
tional team leaders were conducted, along 
with surveys ofthe board and functional team 
members to collect information on the oper- 
ations ofthe partnership and its contribution 
to the development and growth of school-to- 
career in the Kansas City area. 

Findings from the evaluation are summa- 
rized below.^ 

• The Worth of School-to-Career-Related Activi- 
ties (Impact on Students): In analyzing the sur- 
vey and school records data for high school 
students, evaluators grouped students by 
whether they were more or less active in 
school-to-career activities (based on their 
survey responses). Forthe full sample of high 



school students, no statistically significant 
relationships were found between school-to- 
career participation and any of the six indi- 
cators of school outcomes examined. In one 
high school, evaluators found that levels of 
school-to-career participation related posi- 
tively to postsecondary plans, GPA, atten- 
dance, and the students' rating of school 
effectiveness in school-to-career areas. 

Students' level of school-to-career participa- 
tion related to how they felt about their 
schools’ effectiveness in areas associated 
with school-to-career. The evaluators con- 
cluded that certain conditions— having to do 
with the integration of school-to-career 
activities within a school, the depth and 
breadth of implementation, and staff buy-in- 
may contribute to the impact of school-to- 
career. These findings suggest that greater 
impact may be seen when school-to-career 
activities and approaches are more fully 
implemented. 

At the middle school and elementary school 
levels, evaluators found that staff rarely sep- 
arated school-to-career from other school 
practices. However, teachers and adminis- 
trators believed that school-to-career activi- 
ties helped make school relevant, motivated 
young people to finish school, and taught 
behaviors necessary for adult success. 

Worth or usefulness of the BE^: School-to- 
Career Partnership: The evaluators found that 
BE^: STCP had helped to create a “school-to- 
career presence'' in the bi-state region that 
provided visibility, support, and advocacy for 
school-to-career. The intermediary was 
found to facilitate communication between 
the business and education communities and 
to provide a regional infrastructure for link- 
ing people concerned about the education 
and development of youth. The intermedi- 
ary also increased grant success for school- 
to-career programs (both for its parent 
organization and local schools and partner- 
ships) and provided legitimacy for the Learn- 
ing Exchange's involvement in other initia- 
tives (such as workforce development). 



UNITE-LA, Los Angeles, California: Given 
the very large district within which it works, 
UNITE-LA has sought from the beginning to 
have a systematic view of both its work and of 
evaluations of that work. With over 725,000 
students in the LA Unified K-12 system, the 
intermediary’s priority has been to involve as 
many people as possible in school-to-career, 
rather than to focus all its resources on imple- 
menting a few school-to-career activities on a 
significant scale. 

UNITE-LA's overarching goal has signifi- 
cant implications for measuring the impact of 
its work. The intermediary has an ongoing, 
multi-year, external evaluation that aims to 
assess system building efforts largely in terms 
of growth in participation and knowledge of 
school-to-career, and that examines whether 
this growth can be attributed to UNITE-LA. In 
addition, the evaluators are also addressing 
the question of whether school-to-career 
makes a difference by assessing the impact of 
Career Academy participation on school per- 
formance outcomes.^ 

Multi-phase surveys have been used as 
one strategy for assessing growth in school-to- 
career participation. Evaluators began by sur- 
veying a large sample of seniors during the 
1998-99 school year to establish a baseline of 
the frequency and intensity of school-to-career 
involvement and to capture student percep- 
tions of school-to-career activities. A follow-up 
survey administered to a sample of seniors in 
spring 2001 examined the extent of growth in 
activities. A comparison of results from the 
two surveys found few large differences in stu- 
dent responses. However, there were small 
increases in the percentage of seniors who had 
participated in work internships, job shadows, 
and Career Academies, and a larger increase 
(12 percent) in the proportion of students 
who had participated in service learning. 

An effort was also made to analyze the 
impact of UNITE-LA’s grant funding in local 
areas within the district. Interviews and sur- 
veys asked educators and business partners 
about their level of involvement in UNITE-LA 
activities, whether they had received grant 



money, and what they thought about UNITE- 
LA events and campaigns. The questions tried 
to explore the extent to which people attrib- 
uted school-to-career activities to UNITE-LA, 
but evaluators found less attribution to 
UNITE-LA than the intermediary expected. 
This result may reflect the large number of fac- 
tors that contribute to change within a school 
and the fact that many people may feel own- 
ership over helping to create something new. 

Two evaluations of the Academy model 
have been undertaken in an effort to assess 
the impact of a more intensive school-to- 
career initiative on academic outcomes. An 
early evaluation compared school perform- 
ance outcomes for participants in five Acade- 
mies to those for other students in the same 
schools. This study found that Academy stu- 
dents were somewhat more likely to enroll in 
higher-level math classes and had somewhat 
better attendance rates and lower dropout 
rates than their non-Academy peers. 

A second evaluation of the Academy 
approach used a more rigorous, matched- 
comparison-group design that compared the 
performance of Academy students to demo- 
graphically similar LAUSD students not par- 
ticipating in school-to-career programs while 
in high school. (Statistical modeling was used 
to "control" for factors that may have an 
impact on student achievement apart from 
school-to-career participation.) Survey data 
revealed that Academy participants were 
more likely to aspire to postsecondary edu- 
cation and to enroll in postsecondary educa- 
tion once they graduated than non-Academy 
students. Eleventh-grade Academy students 
were also more likely to pass Algebra II (or a 
similar high-level math course), which is 
required for admission to the California State 
or University of California systems. This 
recently completed evaluation found that 
Academy students achieved slightly higher 
standardized test scores in reading and, to a 
lesser extent, in social studies than their non- 
Academy counterparts. No systematic differ- 
ences in test scores were found for other 
areas of the Stanford 9 exam. There were no 



statistically significant differences between 
Academy and non-Academy students on 
attendance or grade point average/ 

PITFALLS AND CONTINUING 

Challenges in Evaluating the 
Work of School-to-Career 
Intermediaries 

Caution and patience are needed when 
looking at changes in school 
performance indicators for school-to- 
career participants. 

There is a great deal of interest on the part of 
funders, educators, business partners, and 
intermediaries in having evidence on whether 
school-to-career activities and interventions 
make a difference in how young people do in 
school and prepare for their future. Schools 
routinely collect data on attendance, grade 
point average, standardized test scores, 
dropout and graduation rates, and the aca- 
demic level of courses taken, and this infor- 
mation can be analyzed to assess the impact 
of school-to-career participation. 

However, such an analysis has to be care- 
fully designed. A major caution echoed by sev- 
eral intermediary directors is not to expect sig- 
nificant behavior change from minimal 
intervention. Therefore, an initial step in ana- 
lyzing school outcomes is to determine the 
school-to-career intervention or level of par- 
ticipation that would be intensive enough to 



Assessing Whether a School-to-Career Intervention is Intensive 
Enough to Impact School Performance 



Randy Wallace with the Tulare County Workforce Coalition suggests 
that school-to-career intermediaries assess school performance indicators 
for interventions that provide youth with three or more significant experi- 
ences that go beyond their day-to-day interaction with a classroom teacher 
and peers. Such experiences can include work internships, student govern- 
ment, community/service learning, academy leadership team, and other 
leadership opportunities. Based on his organization’s evaluations and findings 
from other student surveys, he believes that opportunities such as these 
make young people more optimistic and motivated to learn and help to 
make them better decision-makers. Career Academies and other efforts to 
create smaller groups within high schools enable more students to partici- 
pate in such leadership roles and opportunities for learning outside of school. 



yield a measurable impact on students' school 
performance. If less-intensive interventions are 
examined, the analysis will likely conclude— 
inaccurately— that the school-to-career activi- 
ties made no impact because no changes will 
be seen in student performance. An impact on 
students’ level of interest or motivation may 
exist but not yet to the point it affects school 
performance.® As a director pointed out, one 
or two job shadowing experiences will not 
change students’ grades or attendance. Sever- 
al intermediaries who have examined school 
outcome indicators for school-to-career par- 
ticipants reported finding more positive differ- 
ences for students who were more involved in 
these activities. (See box, "Assessing Whether a 
School-to-Career Intervention is Intensive Enough 
to Impact School Performance.") 

How much school-to-career participation 
or intervention is enough to warrant changes 
in school outcome indicators? No one has a 
definitive answer, and identifying the students 
who have had the more intensive experiences 
can be challenging. The Boston Private Indus- 
try Council and the Philadelphia Youth Net- 
work have learned firsthand that developing 
strategies for identifying individual students 
based on their level of school-to-career 
involvement can require significant invest- 
ment from and close work with the school dis- 
trict’s data or evaluation office. 

Involvement in a specific program is 
sometimes used as a way to identify students 
who have had (or are likely to have had) more 
intensive school-to-career experiences. 
UNITE-LA’s evaluators have used Career 
Academy participation as a proxy for inten- 
sive school-to-career involvement. In a case 
study of 1 5 high schools, their evaluators also 
separated the high schools into "high imple- 
mentation" and “other" school-to-career cate- 
gories based on the level of school-to-career 
programming within the school. Linking 
Learning to Life is preparing to examine 
school outcome indicators for students in spe- 
cific interventions that are more intensive 
(e.g., supportive employment, a vocational 
rehab program) rather than look at students 




who may have experienced one or more job 
shadow, work internship, or other communi- 
ty-based learning opportunity. 

“Compared to who?” is another major 
issue in framing an analysis of school outcome 
indicators. Some intermediaries have found 
positive changes over time in school-wide 
attendance and graduation rates, test scores, 
dropout rates, and GPA. The downside to this 
approach is that any positive impact of school- 
to-career activities is likely to be diluted when 
examining a school's entire student body, 
given the wide range in school-to-career par- 
ticipation levels. In addition, there is no way to 
separate out the impact of school-to-career 
interventions from other simultaneous events 
and factors, such as reform efforts and 
changes in administration.^ 

Some of the same concerns hold true for 
evaluation efforts that look at changes in atten- 
dance, dropout rate, behavior issues, grades, 
etc. for a group of participants in a specific 
school-to-career intervention. It is not possible 
to attribute any positive changes to the inter- 
vention alone: schools are very fluid settings, 
with many factors typically changing at the 
same time. However, positive changes can be 
used to promote the promise of school-to- 
career and inform the community that school 
outcomes are improving as part of the reform 
efforts in a school or district— as long as the 
audience is aware that no single effort 
deserves all the credit for creating the change. 

Matched-comparison groups of students 
in the school who are not participating in the 
school-to-career initiative can help isolate the 
impact of an initiative on participants' school 
outcomes. Some effort is made to control for 
demographics and entering academic levels, 
making this is a better approach than com- 
paring Academy participants, for example, to 
the entire remaining school population. How- 
ever, matched-comparison groups are tricky: 
students who choose to participate in more 
intensive school-to-career activities differ, by 
definition, from those who do not choose to 
participate. Motivation is hard to measure and 
control for.^° 



School-to-career interventions also need 
to be in place long enough to be fully imple- 
mented before their impact is evaluated. In 
the first year or two of implementing a new 
Academy or other school-to-career interven- 
tion, often some pieces are not yet in place, 
limiting the power of the intervention. 

A final challenge in using school records 
to assess the impact of school-to-career on 
student performance is getting the data. 
While schools routinely collect performance 
and outcome data on all students and they are 
partners in the school-to-career efforts being 
evaluated, getting data in a useable form is 
often a difficult task that takes a lot of time. 
School districts need to be prepared to com- 
mit resources for the data entry and pro- 
gramming required to identify students par- 
ticipating in the school-to-career intervention, 
work with evaluators to create an appropri- 
ate comparison group (if this approach is 
being used), and pull out the appropriate stu- 
dents' records in a format useable for statisti- 
cal analysis. Evaluation efforts often get 
delayed during this stage of data negotiation 
and manipulation. 

Intermediaries may want to consider 
strategies for building the capacity of school 
districts to analyze school records data them- 
selves. As described above, the Tulare Coun- 
ty Workforce Coalition in California has been 
working with an outside evaluator and the 
school districts with which the coalition part- 
ners to conduct some analysis of school per- 
formance outcomes. Through this work, the 
coalition is hoping to build the districts' capac- 
ity to analyze the impact of different initia- 
tives. One district involved in a recent evalua- 
tion effort plans to continue an analysis that 
an evaluator started. 

Assessing the impact of school-to-career 
intermediaries on the “community” and 
their success in “system building” is also 
challenging because it is difficult to figure 
out what to measure. 

While school records are often the source for 
assessing the impact of school-to-career on 
young people, the intermediary's impact on 



the community or its effectiveness in system 
building are much harder to address because 
there is no clear source of information or evi- 
dence to answer this question. Several inter- 
mediaries have made an effort to document 
the extent to which “school-to-career” systems 
have been put into place and the role of their 
organizations In creating and supporting these 
systems. The first question has been easier to 
obtain "hard” evidence on than the latter. 

The type of “system building" evidence 
that intermediaries have looked at Includes 
trends in partner and student participation, 
greater awareness of school-to-career, and 
funding sustainability. Questions that have 
been examined include: 

• To what extent have the numbers of partici- 
pating students, teachers, and employers 
Increased? What Is the "repeat customer" 
rate? In other words, do participants of all 
types choose to participate again In the same 
or some other school-to-career activity? 

•Are key players— such as district superin- 
tendents and major employers— at the part- 
ner “table” and do they remain Involved? 

• Has the Intermediary secured additional 
external resources to diversify its funding 
base? Is local financial support for the inter- 
mediary’s work growing? 

•To what extent have education partners 
Implemented policies and resource decisions 
that support school-to-career? 

•Are there overall Improvements In atten- 
dance, graduation rates, and academic per- 
formance in the district or specifically targeted 
schools (with recognition that school-to- 
career may be one of several factors con- 
tributing to change)? 

• Because of the activities of the intermediary, 
has local awareness grown about school-to- 
career and how school-to-career can benefit 
young people? 

• Have federal, state, and local categorical pro- 
grams providing education and training serv- 
ices to youth In the community begun work- 
ing In a more coordinated fashion? 

A smaller number of intermediaries have 
used surveys and focus groups to get feedback 



from partners on perceptions ofthe interme- 
diary's contribution to the community. Such 
surveys tend to ask about the extent to which 
the intermediary has created linkages, whether 
these have helped build partners' capacity, and 
whether having a regional partnership is help- 
ful to creating greater connections among 
youth, youth-serving organizations, employers, 
and schools. These are difficult questions to 
phrase in such a way that there is confidence 
that a variety of partners will understand the 
question in the same way. The BE^: School-to- 
Career Partnership in Kansas City has done sur- 
veying along these lines. To access some survey 
instruments used in the BE^ evaluation, go to: 
WWW. intermediarynetwork. org. 

Attribution of the existence and growth 
of school-to-career activities to an intermedi- 
ary has been difficult to demonstrate. A major 
Issue Is defining school-to-career activities or 
Interventions as separate from other efforts 
going on within schools. Even when this Is fea- 
sible, an intermediary's role In helping start or 
support an activity or program Is not always 
clear to participants. Survey efforts undertak- 
en as part of evaluations in both Kansas City 
and Los Angeles found a relatively small per- 
centage of respondents either aware of the 
intermediary or attributing specific activities 
to the intermediary (although respondents 
often recognized the importance of school-to- 
career activities). 

This is not surprising considering that indi- 
vidual schools play a significant role In what 
actually gets put Into place, and school-to- 
career activities are often merged with other 
initiatives. Some school-to-career activities fit 
with good educational approaches and school 
reform efforts, and are not necessarily labeled 
as a “school-to-career Intervention.” Students 
would likely not be aware of an Intermediary's 
role unless the organization's staff were direct- 
ly Involved in administering the program. This 
also holds true to some extent for participat- 
ing teachers and employers. Schools may get 
most of the credit for a school-to-career inter- 
vention if the program runs out of the school, 
which is generally the case. 



Intermediaries are also interested in 
knowing whether they are choosing the most 
powerful interventions or activities for invest- 
ing their energies and resources. What type of 
data is available to determine whether the ini- 
tiative an intermediary has adopted has had a 
bigger or better impact than another option? 

Despite the difficulty in demonstrating 
whether “system building” is occurring, it is 
important to identify outcomes consistent with 
the goal of system change. If these are taken off 
the table, evaluation efforts will likely focus on 
the impact of school-to-career activities on 
young people’s school performance and behav- 
ior. While these are important goals, if they are 
the only objectives, the focus of the work could 
shift solely to making immediate changes on a 
small number of students and not on system 
and capacity building. To repeat what Richard 
Tulikangas of Linking Learning to Life says, 
“What gets measured, gets done." 

There is a universal desire for longer- 
term follow-up on what happens to 
youth after high school— but who should 
track this type of data? 

Many intermediary staff members expressed a 
desire for longer-term follow up on students 
who participate in school-to-career activities 
during high school. Intermediaries, their fun- 
ders, and partners all want to know whether 
these young people transition to postsec- 
ondary education, if the focus of their postsec- 
ondary studies relates to earlier school-to- 
career activities, whether they complete 
college, and if they transition into employment 
in a related career area. However, no single 
institution is the most logical or best-positioned 
choice to do this type of follow-up. While high 
school guidance counselors usually know about 
a student's plans to attend college at the time 
of graduation, school districts do not typically 
track students to confirm that they actually 
enrolled, what they majored in, or whether 
they graduated. Colleges track their own stu- 
dents while they are enrolled, but they do not 
typically report back to high schools. 

Tracking people over time to gather 
either self-reported information or data col- 



lected by another institution (such as a col- 
lege) can be expensive and difficult, especially 
when young people disperse widely to attend 
colleges beyond their home area. Given the 
cost and the absence of a single organization 
or institution identified as responsible for this 
type of data, longer-term data collection 
remains on a “wish list" in most communities. 

Additional resources and greater capaci- 
ty for data collection are needed before 
longer-term follow-up data can be more wide- 
ly used as evidence of the impact of school-to- 
career. A few Intermediary Network mem- 
bers have undertaken small-scale follow-up 
efforts that offer some strategies for doing 
longer-term follow-up. (See box, "Efforts to Do 
Longer-Term Follow-Up.”) 

Unemployment Insurance data is a 
resource that can be used to look at specific 
groups of people to determine the extent to 
which they are employed, what sector they 
are employed in, and the average salary/ 
wage. This data is reported to state govern- 
ments by all employers for the purposes of 
determining unemployment benefits. As such, 
it is a powerful resource for post-program fol- 
low-up concerning employment. However, 
accessing the data requires the collection of 
individuals' Social Security numbers and state 
cooperation for accessing usable data. These 
conditions are not typically easily met. 

The Philadelphia Youth Network is work- 
ing on a plan to use Unemployment Insurance 
data to gather post-program employment 
information for its work-based learning par- 
ticipants. These efforts have been facilitated 
by PYN's access to data as the fiscal and 
administrative entity for Workforce Invest- 
ment Act youth monies in the Philadelphia 
area. PYN will piggy-back this effort on a lon- 
gitudinal study being done by the School Dis- 
trict of Philadelphia, the Philadelphia Educa- 
tion Fund, and the University of Pennsylvania. 

Intermediaries find it challenging to 

identify hard benefits to employers 
beyond the “feel-good stuff.” 

The lack of long-term follow-up makes it very 
difficult for intermediaries to demonstrate 



direct, tangible benefits to participating 
employers beyond the general “feel good" 
benefit of contributing to the community and 
helping young people. Without tracking 
school-to-career participants beyond the end 
of high school or the beginning of college, it is 
very hard to demonstrate whether employ- 
ers gain a larger, more qualified pool of appli- 
cants as a result of school-to-career. Unem- 
ployment Insurance data is a resource for 
tracking whether young people end up work- 
ing in a field to which they were exposed 
through school-to-career activities, but, as dis- 
cussed above, it has not been used to any 
great degree. 

In intensive youth apprenticeship pro- 
grams, such as those overseen by the Fox 
Cities Alliance for Education in Appleton, Wis- 
consin, intermediary staff know whether par- 
ticipating employers have hired students who 
were in the youth apprenticeship programs. 
Since the number of participating students 
and employers is relatively small, it is easy for 
intermediary staff to obtain this information 
and use it to demonstrate direct benefits 



Efforts to Do Longer-Term Follow-Up 



The Workplace Learning Connection in Cedar Rapids, Iowa, is working 
with the local community college to analyze the extent to which school-to- 
career participants continue to postsecondary education related to their 
school-to-career activities. The community college is analyzing which 
students participated in school-to-career activities while in high school and 
whether there is a connection between a student's focus of study and the 
career or work-related activities he or she did in high school. The Workplace 
Learning Connection hopes to reach out to additional students who attend 
other colleges through their last known address. 



The PhiladelphiaYouth Network hired an opinion research firm to do a 
telephone survey of recent graduates, including students who had participat- 
ed in work-based learning and alumni who had not. (Students were called 
during the October after their graduation.) Youth who had participated in 
work-based learning were more likely to be working in their chosen career 
area and more positive about how their school experience prepared them 
for the world of work than their peers who had not participated in work- 
based learning. The Boston Private Industry Council does a follow-up 
survey with Boston Public School graduates in the fall after graduation to find 
out the extent to which students are in college and are working in their cho- 
sen field. 



when recruiting new employers. 

Rip Rowen from the Capital Area Train- 
ing Foundation in Austin, Texas, suggests 
expanding this type of data collection effort, 
and asking employers to collect information 
on whether the young people they hire have 
been involved in school-to-career activities. 
He hopes to encourage business partners to 
ask the young people they hire whether they 
have participated in a work-based learning 
experience in the industry, became familiar 
with the career area by participating in an 
Academy, or found out about the employer 
through career exploration, job fair, or job 
shadowing. If employers' capacity to collect 
this type of information could be developed, 
it would provide some data on benefits to 
employers and young people. 

Evidence gathering and evaluation can 
require staff and resources that school- 
to-career intermediaries do not always 
have, but some have found inexpensive 
ways to get help with evaluation. 

Collecting data and analyzing evidence 
beyond tracking the numbers of partners and 
participants requires a level of staffing that 
intermediaries often lack. BaySCAN, which 
regularly convenes school-to-career partner- 
ship and intermediary directors in the Bay 
Area of California, reports that the desire for 
evaluation and assessment comes up all the 
time in meetings, but the capacity to pursue 
information and evidence lags significantly 
behind the desire to have it. BaySCAN could 
easily work with partnerships to develop an 
agreed-upon list of important indicators to 
collect information on, but no one had the 
staff or money to regularly gather the data. 

This lack of staff power is one of the rea- 
sons why it can be difficult to get evidence 
gathering towards the top of an organization's 
“to-do" list. Consequently, intermediaries 
sometimes hire an outside evaluator with 
expertise in data collection and evidence who 
can devote attention to evidence gathering. 
However, such evaluations can be expensive. 

Some intermediaries have found ways to 
get outside evaluation help cheaply. As noted 




above, MY TURN has successfully tapped into 
sociology departments at local colleges and 
universities to find inexpensive evaluators. 
Most of these evaluators are graduate stu- 
dents (and sometimes professors) who are 
looking for real world programs to evaluate. 
This approach has enabled her to have an 
outside evaluation for many of MY TURN'S 
programs. 

The Tulare County Workforce Coalition 
has worked In partnership with an outside 
evaluator to help keep evaluation costs down. 
The coalition takes the lead In negotiating 
with school districts for data access, and it 
works with the evaluator and district to 
determine how the data should be analyzed. 
The evaluation work Is also limited to those 
school districts that have the capacity to pro- 
vide school records data and focuses on data 
that the schools already collect (although sur- 
veys and focus groups supplement the school 
records analysis). 

Several Intermediaries have also found 
cost-reducing opportunities to “piggy-back" 
evidence gathering that they want onto other 
efforts being done in the community or by the 
state. 

As noted earlier, “customer satisfaction" 
surveys completed by teachers, students, and 
employers are relatively easy to develop and 
administer. Intermediary staff report that this 
type of feedback provides useful Information 
for continuous Improvement efforts.''’' The 
experience of Network members suggests that 
this type of evidence gathering could be done 
by all school-to-career Intermediaries and 
would be worth the investment of staff time. 

Conclusion 

Evaluations of school-to-career Initiatives and 
the contribution of intermediaries have two 
major goals: obtaining data to Inform 
Improvement and securing support and 
recognition for promising practices and good 
performance. Designing and carrying out such 
evaluations is a significant technical challenge— 
and also a political one. School-to-career 
Intermediaries need to be strategic about the 



design, timing, and focus of data collection and 
evaluation efforts. Too often, intermediaries 
and their funders and partners want to see 
too much, too soon. The expected outcome 
has to match the intensity of change or “input” 
the young people have experienced. Looking 
for impacts on students’ school performance 
indicators before school-to-career interven- 
tions are fully implemented or for students 
with less-intensive school-to-career experi- 
ences is a set-up for failure. 

In considering data collection and evalua- 
tion efforts, intermediaries need to match 
their ambition to their resources and assess 
where the best opportunities are for using 
data to build local support and alliances. It is 
critical to define the purpose and the audience 
for the evaluation work before going too far 
into the design process. Careful planning will 
avoid either over-designing or under-planning 
for the cost and complexity of evaluation 
work. Tracking growth in the types of school- 
to-career interventions offered and the level 
of participation provides credible evidence on 
system building and community support. “Cus- 
tomer satisfaction" surveys of participating 
students, teachers, and employers can pro- 
vide good feedback and useable evidence as 
new activities are put into place and fine- 
tuned. For those with the resources, assess- 
ment of the impact of school-to-career partic- 
ipation on school performance outcomes and 
community “impact” can be carefully done for 
mature, intensive interventions. 

It is clear that school-to-career interme- 
diaries and their funders and partners value 
evidence on the nature and worth of what Is 
being done. Intermediaries that do not collect 
data and evidence can put themselves In a vul- 
nerable position. While the technical and 
political challenges of evaluation can seem 
daunting, intermediaries need to look toward 
building evaluation efforts as they build a 
school-to-career system. The goal should be 
to design and implement a data collection and 
evaluation effort that will be useful, credible, 
and doable, given the nature of local school- 
to-career efforts. 



Appendix 





Youth, District/School, and Employer Participation 
in School-to-Career Activities of School-to-Career Intermediaries 


Intermediary 

Organization 


Number of Youth, Schools, and Employers Participating in School-to-Career Activities 


Business/Education 
Expectations (BE2) 

Kansas City, MO 


Youth Served: 

9,754 directly served (typically involved in a variety of career awareness activities in school and connected to 
the community) 

1,335 brokered placements for job shadows, internships, workplace tours, and apprenticeships 
Participating Districts/Schools: 

65 school districts in two states (primarily in 9 counties with 56 districts); work with 16 high schools, some 
middle schools, and at the elementary level 

192 teachers from 36 high schools placed in summer externships or business rotations 

Participating Employers: 221 active (defined as hosting job shadows or field trips, and/or participating in 
career fairs, mock interviews, or workplace tours); 64 employers (some also counted above) hosted 
teacher externships 


Capital AreaTraining 
Foundation 


Youth Served: 

450 students directly placed into summer youth program involving work placements (additional 2,150 


Austin, TX 


indirectly placed) 

1,000 students in job shadows (mostly indirectly placed) 

Employer/career-related speakers reach students in 500 classes 
Participating Districts/Schools: 

14 school districts and 30 schools (mostly high schools) within these districts (half the schools are deeply 
engaged) 

150 teachers participated in summer externships or institutes 

Participating Employers: 400 active employers (defined as those who participate in one or more of the follow- 
ing: internships and rotations, career awareness (speakers, job shadows, and tours), educator professional 
development, and/or involvement in industry clusters 

(Numbers are approximations) 


Fox Cities Alliance 
for Education 


Youth Served: 

20,677 student contacts (includes double counts if youth participated in more than one activity/service; 


Appleton, Wl 


most contact is through in-school presentations, use of intermediary-developed resources, and visits to 
Career Exploration Center) 

Total includes 805 students in job shadows and 60 students in youth apprenticeship programs 

Districts/Schools participating: 9 districts, working with 1 1 high schools and some middle and elementary 
schools 

Participating Employers: 325 have participated at varying levels; 51 training students as part of youth appren- 
ticeship and certified co-op program 


Linking Learning 
to Life 


Youth Served: 

2,500 student contacts (includes double counts if youth participated in more than one activity/service; most 


Burlington, VT 


contact involved career exploration activities) 

Total includes: 485 in job shadows, 135 career worksite visits and tours, 40 in work-based learning experi- 
ences, and 120 special education students in a vocational rehab program 

Districts/Schools Participating: 

1 district with 6 elementary, 2 middle, and 1 high school, plus 1 regional tech center and a couple of alterna- 
tive programs 

Participating Employers: 253 involved in community-based experiences (employers to schools and youth to 
employers); 50 involved with vocational rehab program (some double counting) 



Metro Vision School-to- 
Career Partnership 

New Orleans, LA 


Youth Served: 

4,782 job shadows 

142 Internships (connected to Academies) 

Indirect services to students In 118 Academies/Career Clusters that MetroVislon has helped to put into 
place and 789 students served by out-of-school youth consortia (42 partners) 

Participating Schools/ Districts: 

Work with a total of 7 school districts and 36 high schools and 14 middle schools within these districts 
Working with 1 3 high schools on Academy implementation and whole school change 
91 teachers in internships (over 200 a year in previous years) 

Participating Employers: 462 hosted job shadows or internships and/or participated in career-related activi- 
ties in schools 


Middle Rio Grande 
Business and Education 
Collaborative 


Youth Served: 

100 students placed in job shadows and 500 indirect placements 


Albuquerque, NM 


1,500 students participated in regional career fair leading to direct and indirect placements and 1,020 stu- 
dents working with Path Finder software to identify placements 

Participating Schools/ Districts: 

Work with 10 school districts, which include 30 high schools, an alternative high school, and 45 middle 
schools; most work at the high school level with some Pathfinder connection work at the middle school level 

200 teachers in work-based experiences 

140 teachers involved in Web-based training and professional development activities 

Participating Employers: 300 directly active employers, plus another 800 involved in career and hiring fairs, 
indirect job shadow placements, conferences, etc. 


MY TURN, Inc. 


Youth Served: 9,800 


Brockton, MA 


Participating Schools: 1 high schools 

Participating Employers: 180 active employers (defined as involved in job shadows, internships, guest speakers) 


Philadelphia Youth 
Network 


Youth Served: 

1,1 16 direct placements in work-based learning and subsidized and unsubsidized employment 


Philadelphia, PA 


1, 500 students in job shadows 

6,192 brokered placements/services (includes WlA-funded after-school and summer academic enrichment 
and work experience) 

Participating Districts/Schools: 

Work with all the schools in one large school district (total of 44 high schools, plus some middle and elemen- 
tary schools) plus Archdiocesan, private, and charter schools 

1 ,200 teachers involved in professional development, service learning, integrating education into workplace 
experiences (some are advisors to brokered programs) 

Participating Employers: 529 active employers (defined as those who provide work-based experiences for 
youth and partners who help to set policy); does not include employers involved only in job shadow day 


Pittsburgh Technology 
Council 


Youth Served: 

2,000 indirect placements in a range of work-based learning experiences 


Pittsburgh, PA 


23 students in Manufacturing Pathways Initiative internships (slated to greatly increase in summer 2002) 
Districts/Schools Participating: 

Work with a total of 13 counties, with about 150 districts; direct schooi-to-career programmatic efforts tar- 
geted at 9 counties; work primarily at the high school level 

3 districts and 1 vocational/technical school participating in MPI, with hope to expand to 65 districts and 12 
voc tech 

One district and 1 10 teachers participating in information technology literacy professional development 
effort, which will be expanded to 10 districts in 2002-03 

Participating Employers: 10 in MPI, with plan for 200; 20 on steering committee; 1,600 employers involved 
with council's capacity as a trade association 



Tulare County 
Workforce Coalition 


Youth Served: 

7,455 contacted directly (events, targeted projects) 


Tulare County Office 
of Education 


17,545 students received information and assistance through classroom and program materials and support 


Visalia, CA 


Participating Districts/Schools: 9 districts, with 1 9 comprehensive high schools within these districts 
Participating Employers: 955. 


UNITE-LA 


Youth Served: 


Los Angeles, CA 


400 directly placed into internships and additional 1,300 indirectly placed 

8.000 students in job shadows 

1 3.000 students visited college campuses as part of college campaign 

Estimated 25-50% of youth in the LA Unified School district touched in someway by intermediary facilitators 
working with local areas (sub-districts) within the district 

Participating Districts/Schools: 

One very large school district: LA Unified School District, with 725,000 students in K-12 system 
Help to support 65 Career Academies, with about 10,000 students 

Placed 600 educators in externships last winter, with over 1,000 involved in different activities 
60 principals were executives for a day; about 85 business people were principal for the day 
Participating Employers: 

1.000 businesses involved in some capacity 


The Workplace 
Learning Connection 

Cedar Rapids, lA 


Youth Served: 

1,915 job shadows 
150 (unpaid) internships 

24 middle school students in career explorer camp 
Districts/Schools Participating: 

33 districts, including 45 high schools and a total of 60,000 students in K-12 (WLC does some work at middle 
and elementary level) 

50 teachers placed in summer externships 

Participating Employers: 700 

(tours of business by educators and youth not included In these numbers) 




Note: Information was collected for a one-year time period (usually the 2000-2001 school year). Staff of the Bay 
Area School-to-Career Action Network (BaySCAN) were also Interviewed, but the organization operates like an inter- 
mediary for intermediaries. It does not collect data on the numbers of youth, schools, and employers participating In 
BaySCAN activities. Interviews were conducted with staff members at all the organizations in this chart except the 
Middle Rio Grande Business and Education Collaborative. 



Notes 



^ The Intermediary Network began in 1998 with 
25 members, with 25 more joining since then. 

^ These tools are available at 
www.lntermedlarynetwork.org or by contact- 
ing New Ways to Work. 

^ These findings come from Making a Difference: 
School-to-Career Evaluation, Tulare County 
Offce of Education, May 2001. The analysis did 
not control for pre-participation differences 
between Academy students and the general 
student population. 

Summary of findings taken from "The Positive 
Effects of Work-Based Learning in Philadelphia," 
prepared by the Philadelphia Youth Network. 

^ Summary findings taken from Conclusions from 
the SET STCP Evaluation Study, by Jacob 
Blasczyk, University of Wisconsin-Madison, 
Center on Education and Work, February 
2001 , 

^ Information for this summary comes from 
UNITE-LA STC Evaluation: Year 3 Interim Report 
(revised) by Public Works, October 2001 and 
the Executive Summary of the 1999-2000 
Academy Study (also by Public Works, Inc.). 

The evaluation of school-to-career in the LA 
Unified district is extensive, complex, and now 
tied In part to the evaluation of school-to- 
career in California. Only part of the evaluation 
effort is summarized here. 

^ Summary findings taken from Evaluation of the 
UNITE-LA School-to-Career Partnership: PLUS 
Evaluation ofLAUSD Career Academies, 
prepared by Public Works, Inc., March 2002. 



® This is one reason to include surveys along with 
any analysis of school outcome indicators. Sur- 
veys can help capture self-reported changes in 
motivation and level of interest even if no 
impact on attendance or grades can be seen 
yet. 

^ The student body also changes over time, so 
positive or negative findings may reflect a differ- 
ent group of students rather than any changes 
in what is happening in the school. 

An experimental design that randomly assigns 
students who apply to participate to either the 
school-to-career intervention or a “control” 
group that does not receive the intervention is 
the cleanest way to create a true comparison 
group, but random assignment experiments are 
rarely done in school settings. An exception is 
the Manpower Demonstration Research Cor- 
poration’s Career Academy evaluation current- 
ly underway in nine schools across the country. 
In a March 2000 report on the evaluation, 
MDRC noted that the Career Academies sub- 
stantially improved high school outcomes 
among students at high risk of dropping out, 
and, among students least likely to drop out, 
the Career Academies increased the likelihood 
of graduating on time. The Career Academies 
were not found to improve students’ standard- 
ized math and reading achievement test scores, 

'''' Given a good facilitator, focus groups are rela- 
tively easy to do and can provide useful infor- 
mation. 



The School-to-Work Intermediary Project 



The School-to-Work Intermediary Project is designed to strengthen 
and raise the public profile of local organizations that connect schools, 
workplaces, and other community resources. 



www.intermediarynetwork.org 



Jobs for the Future 

88 Broad Street, 8th Floor 
Boston, MA 02110 
(617)728-4446 



New Ways to Work 

103 Morris Street, Suite A 
Sebastopol, CA 95472 
(707)824-4000 




